Method and apparatus for implementing a digital graduated filter for an imaging apparatus

ABSTRACT

A digital graduated filter is implemented in an imaging device by combining multiple images of the subject wherein the combining may include combining different numbers of images for highlights and for shadows of the subject. The imaging device may present a user with a set of pre-defined graduated filter configurations to choose from. A user may also specify the direction of graduation and strength of graduation in a viewfinder. In an alternative implementation, combining may include scaling of pixels being added instead of varying the number of images being combined. In an alternative implementation, the combining of multiple images may include combining a different number of images for highlights of the subject than for shadows of subject.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/811,171 filed Nov. 13, 2017 which is a continuation of U.S. patentapplication Ser. No. 14/861,731 filed Sep. 22, 2015, which issued asU.S. Pat. No. 9,826,159 on Nov. 21, 2017, which is a continuation inpart of U.S. patent application Ser. No. 14/679,551 filed on Apr. 6,2015, which issued as U.S. Pat. No. 9,392,175 on Jul. 12, 2016, which isa continuation of U.S. patent application Ser. No. 13/653,144, filed onOct. 16, 2012, which was issued as U.S. Pat. No. 9,001,221 on Apr. 7,2015, which is a continuation of U.S. Pat. application Ser. No.13/442,370, filed on Apr. 9, 2012, which issued as U.S. Pat. No.8,922,663 on Dec. 30, 2014, which is a continuation of U.S. patentapplication Ser. No. 12/274,032, filed on Nov. 19, 2008, which issued asU.S. Pat. No. 8,154,607 on Apr. 10, 2012, which is a continuation ofU.S. patent application Ser. No. 11/089,081, filed on Mar. 24, 2005,which issued as U.S. Pat. No. 8,331,723 on Dec. 11, 2012, which claimsthe benefit of U.S. Provisional Application Ser. No. 60/556,230, filedon Mar. 25, 2004, the contents of each of which are incorporated byreference herein as if fully set forth .

FIELD OF INVENTION

The present invention generally relates to digital image processing.More specifically, this invention relates to processing of digitizedimage data in order to correct for image distortion caused by relativemotion between the imaging device and the subject at the time of imagecapture, or by optical distortion from other sources. This inventionalso relates to improving image quality, improving color and lightdynamic range, and enhancing images through signal processing.

BACKGROUND

When capturing images, as with a camera, it is desirable to captureimages without unwanted distortion. In general, sources of unwanteddistortion may be characterized as equipment errors and user errors.Examples of common equipment errors include inadequate or flawed opticalequipment, and undesirable characteristics of the film or otherrecording media. Using equipment and media of a quality that is suitablefor a particular photograph may help mitigate the problems associatedwith the equipment and the recording medium, but in spite of this, imagedistortion due to equipment errors may still appear.

Another source of image distortion is user error. Examples of commonuser errors include poor image processing, and relative motion betweenthe imaging device and the subject of the image. For example, one commonproblem that significantly degrades the quality of a photograph is theblur that results from camera movement (i.e. shaking) at the time thephotograph is taken. This may be difficult to avoid, especially when aslow shutter speed is used, such as in low light conditions, or when alarge depth of field is needed and the lens aperture is small.Similarly, if the subject being photographed is moving, use of a slowshutter speed may also result in image blur.

There are currently many image processing techniques that are used toimprove the quality, or “correctness,” of a photograph. These techniquesare applied to the image either at the time it is captured by a camera,or later when it is post-processed. This is true for both traditional“hardcopy” photographs that are chemically recorded on film, and fordigital photographs that are captured as digital data, for example usinga charged couple device (CCD) or a CMOS sensor. Also, hardcopyphotographs may be scanned and converted into digital data, and arethereby able to benefit from the same digital signal processingtechniques as digital photographs.

Commonly used post-processing techniques for digitally correctingblurred images typically involve techniques that seek to increase thesharpness or contrast of the image. This may give the mistakenimpression that the blur is remedied. However, in reality, this processcauses loss of data from the original image, and also alters the natureof the photograph. Thus, current techniques for increasing the sharpnessof an image do not really “correct” the blur that results from relativemotion between a camera and a subject being photographed. In fact, thedata loss from increasing the sharpness may result in a less accurateimage than the original. Therefore, a different method that actuallycorrects the blur is desirable.

In the prior art, electro-mechanical devices for correcting image blurdue to camera motion are built into some high quality lenses, variouslycalled “image stabilization”, “vibration reduction”, or similar names bycamera/lens manufacturers. These devices seek to compensate for thecamera/lens movement by moving one or more of the lens elements; hencecountering the effect of the motion. Adding such a device to a lenstypically makes the lens much more expensive, heavier and less sturdy,and may also compromise image quality.

Accordingly, it is desirable to have a technique that corrects fordistortion in photographs without adding excessively to the price,robustness or weight of a camera or other imaging device, or adverselyaffecting image quality.

An additional limitation of current digital imaging devices is that thedynamic range of the image sensors are not adequate to capture bothshadows and highlights with detail. As a result, many digitalphotographs result in washed out highlights or completely dark shadowsthat are devoid of detail. In traditional film photography theseproblems are experienced less because most types of film have largerdynamic range compared to digital image sensors.

One remedy in dealing with subjects with large dynamic range hastraditionally been the use of graduated filters. These are glass filtersthat are attached in front of a lens to limit the light coming into thelens from certain areas of the subject. These filters in effect compressthe dynamic range of the light coming from the subject. For example, ifa scene includes a dark meadow below and a very light sky above, agraduated filter that limits light going through it in the upper part ofthe image reduces the light intensity for the highlights and“compresses” the light dynamic range of the scene. In this way, bothhighlights, such as the sky, and the shadows, such as the dark meadow,are captured by the camera with detail. Although graduated filters helpin many high-dynamic-range scenes, they are not convenient. A person hasto carry along one or more graduated filters for each lens, and adjustthe orientation of the graduation by rotating the filter every time apicture is taken.

Accordingly, it is desirable to have a technique that enables digitalimaging devices to capture subjects with large dynamic ranges, withoutrequiring use of external graduated filters.

SUMMARY

The present system and method process image data in order to correct animage for distortion caused by imager movement or by movement of thesubject being imaged. In another embodiment, the present invention mayprevent image distortion due to motion of the imaging device or subjectat relatively slow shutter speeds, resulting in a substantiallyundistorted image.

In another embodiment, the system and method measure relative motionbetween the imaging device and the subject by using sensors that detectthe motion. When an image is initially captured, the effect of relativemotion between the imaging device and the subject is that it transformsthe “true image” into a blurred image, according to a 2-dimensionaltransfer function defined by the motion. The system and method determinea transfer function that represents the motion and corrects the blur.

In yet another embodiment, the transfer function is estimated usingblind detection techniques. The transfer function is then inverted, andthe inverted function is implemented in an image correcting filter thatessentially reverses the blurring effect of the motion on the image. Theimage is processed through the filter, wherein blur due to the motion isreversed, and the true image is recovered.

In yet another embodiment, the invention uses the transfer function tocombine consecutive images taken at a fast shutter speed to avoid blurdue to motion between camera and subject that could result from using aslow shutter speed. In still another embodiment, the image sensor ismoved to counter camera motion while the image is being captured.

In yet another embodiment, the invention uses consecutive images takenof the subject to overcome distortions due to limited dynamic range ofthe image sensor. By selectively combining consecutive images, theinvention creates an improved resulting image that renders highlightsand shadows with detail, which would otherwise fall outside the dynamicrange of image sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a portion of memory having memory locations wherein elementsof a recorded image are stored.

FIG. 2 is a portion of memory having memory locations wherein elementsof a deconvolution filter are stored.

FIG. 3 is a portion of memory having memory locations wherein therecorded image is stored for calculating the next value of a correctedimage.

FIG. 4 is a functional block diagram of a system for correcting an imagefor distortion using a transfer function representing the distortion,wherein the transfer function is derived from measurements of the motionthat caused the distortion.

FIG. 5 is a functional block diagram of a system for correcting an imagefor distortion using a transfer function representing the distortion,wherein the transfer function is derived using blind estimationtechniques.

FIG. 6 shows a unit for iterative calculation of the corrective filtercoefficients and estimation of the correct image data.

FIG. 7 illustrates support regions of an image r(n,m) and of a transferfunction h(n,m), and the transfer function h(n,m) being applied todifferent parts of the image r(n,m).

FIG. 8 shows a unit for blind deconvolution to calculate the correctimage data.

FIG. 9 is an image of an object being captured on an image sensorwherein pixel values represent points of the image.

FIG. 10 illustrates the effect of moving an imager while capturing animage, resulting in multiple copies of the image being recorded overeach other, causing blur.

FIG. 11 illustrates combining images taken at fast shutter speeds toresult in the equivalent of a final image taken at a slower shutterspeed, but with reduced blur.

FIG. 12 illustrates a system for image blur correction where an imagesensor is moved to compensate for imager movement.

FIG. 13 is an example of an image distorted by movement of the imagerwhen the image was captured.

FIG. 14 is represents the image of FIG. 13 corrected according to thepresent invention.

FIGS. 15A-15B illustrate combining of multiple images to implement anall-digital graduated filter.

FIG. 16 illustrates a plurality of graduated filters available forselection.

FIG. 17 illustrates a way for selection of orientation of the graduatedfilter.

FIG. 18A illustrates how parts of an image may be saturated when adynamic range is larger than what may be captured with the image sensor.

FIG. 18B illustrates how a lower light setting may be used to ensurethat the image is not saturated, but with only a portion of the dynamicrange of the image sensor being captured.

FIG. 18C illustrates combining multiple images to form a final imagethat takes up the full dynamic range of the image sensor and avoidssaturation.

FIG. 18D illustrates the comparison between a case where a large dynamicrange subject saturates the image sensor and one where the same subjectis recorded multiple times and then selectively combined to form a finalimage that is not saturated.

FIG. 19 illustrates scaling of pixel values to create a large dynamicrange to avoids saturation.

FIG. 20 illustrates a process of capturing multiple images of a subjectand selectively decreasing light intensity for by scaling pixel values.

FIG. 21 illustrates a process of selectively combining multiple images.

FIG. 22 is an illustration of a process of capturing multiple images ofa subject and selectively scaling pixel values.

FIG. 23 illustrates a process of selectively capturing multiple images.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present apparatus, system and method will be described withreference to the figures wherein like numerals represent like elementsthroughout. Although hereinafter described as an apparatus, system andmethod of correcting for image distortion due to the shaking of a camerawhen a picture is taken, similar distortions may also be caused by othertypes of imaging equipment and by imperfections in photo processingequipment, movement of the subject being photographed, and othersources. The present apparatus, system and method may also be applied tocorrect for these types of distortions.

Saturation of the imaging device is another problem that results insub-optimal image quality. The present apparatus, system and method maybe utilized to correct for these types of distortions as well.

Although reference is made throughout the specification to a camera asthe exemplary imaging device, the present invention is not limited tosuch a device. As aforementioned, the teachings of the present inventionmay be applied to any type of imaging device, as well as imagepost-processing techniques.

For sake of clarity, a camera may be a stand-alone photographyequipment, as well as a function included in another electronic devicesuch as a smartphone, tablet computer, a wearable electronic device, orother personal communication or personal computing device.

Capturing and recording a photograph, for example by a camera, involvesgathering the light reflected or emanating from a subject, passing itthrough an optical system, such as one or more lenses, and directing itonto a light sensitive recording medium. A typical recording medium intraditional analog photography is a film that is coated with lightsensitive material. During processing of the exposed film, the image isfixed and recorded. In digital cameras, the recording medium istypically a dense arrangement of light sensors, such as a Charge-CoupledDevice (CCD) or a CMOS sensor.

The recording medium continuously captures the impression of the lightthat falls upon it as long as the camera shutter is open. Therefore, ifthe camera and the subject are moving with respect to each other (suchas in the case when the user is unsteady and is shaking the camera, orwhen the subject is moving), the recorded image becomes blurred. Toreduce this effect, a fast shutter speed may be used, thereby reducingthe amount of motion occurring while the shutter is open. However, thisreduces the amount of light from the subject captured on the recordingmedium, which may adversely affect image quality. In addition,increasing the shutter speed beyond a certain point is not alwayspractical. Therefore, undesired motion blur occurs in many picturestaken by both amateur and professional photographers.

The nature of the blur is that the light reflected from a referencepoint on the subject does not fall on a single point on the recordingmedium, but rather it “travels” across the recording medium. Thus aspread-out, or smudged, representation of the reference point isrecorded.

Generally, all points of the subject move together, and the optics ofthe camera and the recording medium also move together. For example, inthe case of a photograph of a moving car, wherein an image of the car isblurred due to uniform motion of all parts of the car. In other words,the image falling on the recording medium “travels” uniformly across therecording medium, and all points of the subject blur in the same manner.

The nature of the blur resulting from uniform relative motion may beexpressed mathematically. In a 2-dimensional space with discretecoordinate indices ‘n’ and ‘in’, the undistorted image of the subjectmay be represented by s(n,m), and a transfer function h(n,m) may be usedto represent the blur. Note that h(n,m) describes the way the image“travels” on the recording medium while it is captured. The resultingimage that is recorded, r(n,m), is given by:

r(n,m)=s(n,m)**h(n,m);   Equation (1)

where * * represents 2-dimensional convolution. The mathematicaloperation of convolution is well known to those skilled in the art anddescribes the operation:

$\begin{matrix}{{r\left( {n,m} \right)} = {\sum\limits_{i = {- \infty}}^{\infty}{\sum\limits_{j = {- \infty}}^{\infty}{{h\left( {i,j} \right)}{{s\left( {{n - i},{m - j}} \right)}.}}}}} & {{Equation}\mspace{14mu} (2)}\end{matrix}$

In the sum operations in Equation (2), the summation limits areinfinite. In practice, the summations are not infinite, since thesupport region of the transfer function is finite. In other words, theregion where the function is non-zero is limited by the time the camerashutter is open and the amount of motion. Therefore, the summation iscalculated for only the indices of the transfer function where thefunction itself is non-zero, for example, from i=−N . . . N and j=−M . .. M.

If the transfer function h(n,m) is known, or its estimate is available,the blur that it represents may be “undone” or compensated for in aprocessor or in a computer program, and a corrected image may beobtained, as follows. Represent the “reverse” of the transfer functionh(n,m) as h⁻¹(n,m) such that:

h(n,m)* * h ⁻¹(n,m)=δ(n,m);   Equation (3)

where δ(n,m) is the 2-dimensional Dirac delta function, which is:

$\begin{matrix}{{\delta \left( {n,m} \right)} = \left\{ {\begin{matrix}1 & {{{if}\mspace{14mu} n} = {m = 0}} \\0 & {otherwise}\end{matrix}.} \right.} & {{Equation}\mspace{14mu} (4)}\end{matrix}$

The delta function has the property that when convolved with anotherfunction, it does not change the nature of that function. Therefore,once h(n,m) and hence h⁻¹(n,m) are known, an image r(n,m) may be putthrough a correcting filter, called a “deconvolution filter”, whichimplements the inverse transfer function w(n,m)=h⁻¹(n,m) and undoes theeffect of blur. Then:

$\begin{matrix}\begin{matrix}{{{r\left( {n,m} \right)}^{**}{w\left( {n,m} \right)}} = {{r\left( {n,m} \right)}^{**}{h^{- 1}\left( {n,m} \right)}}} \\{= {{s\left( {n,m} \right)}^{**}{h\left( {n,m} \right)}^{**}{h^{- 1}({nm})}}} \\{= {{s\left( {n,m} \right)}^{**}{\delta \left( {n,m} \right)}}} \\{{= {s\left( {n,m} \right)}};}\end{matrix} & {{Equation}\mspace{14mu} (5)}\end{matrix}$

and the correct image data s(n,m) is recovered.

The deconvolution filter in this example is such that:

$\begin{matrix}{{\sum\limits_{i = {- N}}^{N}{\sum\limits_{j = {- M}}^{M}{{w\left( {i,j} \right)}{h\left( {{n - i},{m - j}} \right)}}}} = \left\{ {\begin{matrix}1 & {{{if}\mspace{14mu} n} = {m = 0}} \\0 & {otherwise}\end{matrix}.} \right.} & {{Equation}\mspace{14mu} (6)}\end{matrix}$

Because of the property that the deconvolution operation forces theoutput of the convolution to be zero for all but one index, this methodis called the “zero-forcing algorithm”. The zero-forcing algorithmitself is but one method that may be used, but there are others possiblealso, such as the least mean-square algorithm described in more detailbelow.

In order to define a deconvolution filter, the transfer function h(n,m)representing the relative motion between the imager and the subject mustbe derived from measuring the motion, or alternatively by using blindestimation techniques. The inverse function h⁻¹(n,m) must then becalculated and incorporated in a filter to recover a corrected images(n,m). It is possible to determine h(n,m) using sensors that detectmotion, and record it at the time the image is captured.

One embodiment of the present invention includes one or more motionsensors, attached to or included within the imager body, the lens, orotherwise configured to sense any motion of the imager while an image isbeing captured, and to record this information. Such sensors arecurrently commercially available which are able to capture movement in asingle dimension, and progress is being made to improve their accuracy,cost, and characteristics. To capture motion in two dimensions, twosensors may be used, each capable of detecting motion in a singledirection. Alternatively, a sensor able to detect motion in more thanone dimension may be used.

The convolution in Equation (5) may be performed using memory elements,by performing an element-by-element multiplication and summation overthe support region of the transfer function. The recorded image isstored, at least temporarily, in memory elements forming a matrix ofvalues such as shown in FIG. 1. Similarly, the deconvolution filterw(n,m) is stored in another memory location as shown in FIG. 2. Thedeconvolution operation is then performed by multiplying the values inthe appropriate memory locations on an element-by-element basis, such asmultiplying r(n,m) and w(0,0); r(n−1,m) and w(1,0), and so on, andsumming them all up.

Element-by-element multiplication and summing results in theconvolution:

$\begin{matrix}{{y\left( {n,m} \right)} = {\sum\limits_{i = {- N}}^{N}{\sum\limits_{j = {- M}}^{M}{{w\left( {i,j} \right)}{{r\left( {{n - i},{m - j}} \right)}.}}}}} & {{Equation}\mspace{14mu} (7)}\end{matrix}$

To calculate the next element, y(n+1,m) for example, the deconvolutionfilter w(n,m) multiplies the shifted memory locations, such as shown inFIG. 3, followed by the summation. Note that the memory locations do notneed to be shifted in practice; rather, the pointers indicating thememory locations would move. In FIG. 1 and FIG. 3 portions of r(n,m) areshown that would be included in the element-by-element multiplicationand summation, and this portion is the same size as w(n,m). However, itshould be understood that r(n,m), that is the whole image, is typicallymuch larger than the support region of w(n,m). To determine value of theconvolution for different points, an appropriate portion of r(n,m) wouldbe included in the calculations.

The filter defined by Equation (5) is ideal in the sense that itreconstructs the corrected image from the blurred image with no dataloss. A first embodiment calculates the inverse of h(n,m) where h(n,m)is known. As explained above, by making use of motion detecting devices,such as accelerometers, the motion of the imager (such as a cameraand/or the associated lens) may be recorded while the picture is beingcaptured, and the motion defines the transfer function describing thismotion.

A functional block diagram of this embodiment in accordance with thepresent invention is illustrated in FIG. 4, wherein a method 40 forcorrecting image distortion is shown. An image r(n,m) from camera opticsis captured by an imager (step 41) and recorded in memory (step 42).Simultaneously, motion sensors detect and record camera motion (step 43)that occurs while the shutter of the camera is open. The transferfunction representing the motion h(n,m) is derived (step 44), and theinverse transfer function h⁻¹ (n,m) is determined (step 46). The inversetransfer function is applied in a corrective filter (step 48) to theimage, which outputs a corrected image s(n,m) (step 49).

In this and other embodiments that make use of motion sensors torepresent the imager's movement, derivation of the transfer functionfrom motion information (step 44) takes into account the configurationof the imager and the lens also. For an imager that is a digital camera,for example, the focal length of the lens factors into the way themotion of the imager affects the final image. Therefore theconfiguration of the imager is part of the derivation of h(n,m). This isimportant especially for imagers with varying configurations, such asdigital cameras with interchangeable lenses.

In this first embodiment of the apparatus, system and method, aniterative procedure is used to compute the inverse transfer functionfrom h(n,m). The approximate inverse transfer function at iteration k isdenoted as ĥ_(k) ⁻¹(n, m). At this iteration, output of thedeconvolution filter is:

$\begin{matrix}\begin{matrix}{{y_{k}\left( {n,m} \right)} = {{{\hat{h}}_{k}^{- 1}\left( {n,m} \right)}^{**}{r\left( {n,m} \right)}}} \\{= {\sum\limits_{i}{\sum\limits_{j}{{{\hat{h}}_{k}^{- 1}\left( {i,j} \right)}{{r\left( {{n\; - i},{m - j}} \right)}.}}}}}\end{matrix} & {{Equation}\mspace{14mu} (8)}\end{matrix}$

The filter output may be written as the sum of the ideal term and theestimation noise as:

$\begin{matrix}\begin{matrix}{{y_{k}\left( {n,m} \right)} = {{{h^{- 1}\left( {n,m} \right)}^{**}{r\left( {n,m} \right)}} +}} \\{{\left( {{{\hat{h}}_{k}^{- 1}\left( {n,m} \right)} - {h^{- 1}\left( {n,m} \right)}} \right)^{**}{r\left( {n,m} \right)}}} \\{{= {{s\left( {n,m} \right)} + {v_{k}\left( {n,m} \right)}}};}\end{matrix} & {{Equation}\mspace{14mu} (9)}\end{matrix}$

where v(n,m) is the estimation noise which is desirable to eliminate. Aninitial estimate of the correct image may be written as:

Ŝ _(k)(n,m)=ĥ _(k) ⁻¹(n,m)**r(n,m).   Equation (10)

However, this estimate may in general be iteratively improved. There area number of currently known techniques described in estimation theory toachieve this. A preferable option is the Least Mean-Square (LMS)algorithm. A block diagram of a calculation unit 60 which implementsthis method is shown in FIG. 6.

As an initial state, ĥ⁻¹ ₀(n,m) is set to equal ,ur(n,m). Then, thefollowing steps are iteratively repeated:

Step 1, an estimate of the correct image is calculated in a first2-dimensional finite impulse response (2D FIR) filter 62:

Ŝ _(k)(n,m)=ĥ _(k) ⁻¹(n,m)**r(n,m).

Step 2, a received signal based on the estimated correct image iscalculated in a second 2D FIR filter 64:

{tilde over (r)} _(k)(n,m)=Ŝ _(k)(n,m)**h(n,m);

and the estimation error is calculated using an adder 66:

e _(k)(n,m)=r _(k)(n,m)−{tilde over (r)} _(k)(n,m).

Step 3, the inverse transfer function coefficients are then updated inthe LMS algorithm unit 68:

ĥ _(k+1) ⁺¹(n,m)=ĥ _(k) ⁻¹(n,m)+μr(n,m)e _(k)(n,m);

where μ is the step-size parameter.

These steps are repeated until the estimation error becomes small enoughto be acceptable; which value may be predetermined or may be set by auser. As the iterative algorithm converges, the estimated inversetransfer function approaches the correct inverse transfer functionh⁻¹(n,m). The inverse transfer function coefficients are thecoefficients of the deconvolution filter, and the estimate Ŝ(n,m)converges to s(n,m), the correct image, at the same time.

This process may be repeated for the entire image, but it is lesscomplex, and therefore preferable, to find the inverse filter first overa single transfer function support region, then apply it to the entireimage r(n,m).

While the above steps 1-3 are being repeated, a different portion of therecorded image r(n,m) may be used in each iteration. As in FIG. 7, itshould be noted that the recorded image r(n,m) typically has a muchlarger support region than the transfer function h(n,m) that representsthe camera motion. Therefore, the above steps are preferably performedover a support region of h(n,m), and not over the entire image r(n,m),for each iteration.

Although the present apparatus, system and method have been explainedwith reference to the LMS algorithm, this is by way of example and notby way of limitation. It should be clear to those skilled in the artthat there are other iterative algorithms beside the LMS algorithm thatmay be used to achieve acceptable results, and also that there areequivalent frequency domain derivations of these algorithms. Forexample, it is possible to write Equation (1) in frequency domain as:

R(ω₁,ω₂)=S(ω₁,ω₂)H(ω₁,ω₂);   Equation (11)

where R(ω₁,ω₂), S(ω₁,ω₂), and H(ω₁,ω₂) are the frequency domainrepresentations (Fourier Transforms) of the captured image, the correctimage, and the transfer function, respectively, and therefore:

$\begin{matrix}{{S\left( {\omega_{1},\omega_{2}} \right)} = {\frac{R\left( {\omega_{1},\omega_{2}} \right)}{H\left( {\omega_{1},\omega_{2}} \right)}.}} & {{Equation}\mspace{14mu} (12)}\end{matrix}$

To obtain s(n,m) one would calculate S(ω₁, ω₂)as above and take theInverse Fourier Transform, which should be known to those skilled in theart. However, this method does not always lead to well behavedsolutions, especially when numerical precision is limited.

In a second embodiment of the present apparatus, system and method,h(n,m) is not known. This second embodiment uses so-called blinddeconvolution, whereby the transfer function h(n,m) is estimated usingsignal processing techniques. A functional block diagram of thisembodiment is illustrated in FIG. 5, wherein a method 50 for correctingimage distortion according to this embodiment is shown. An image r(n,m)from the optics from a camera is captured (step 51) and recorded inmemory (step 52). Unlike the first embodiment, there are no motionsensors to detect and record camera motion that occurs while the shutterof the camera is open. Instead, the transfer function representing themotion h(n,m) is derived using blind estimation techniques (step 54),and the inverse transfer function h⁻¹(n,m) is determined (step 56). Theinverse transfer function is applied in a corrective filter to the image(step 58), which outputs a corrected image s(n,m) (step 59).

Blind equalization techniques are used to obtain the deconvolutionfilter coefficients. This is also an iterative LMS algorithm, similar tothat used in the first embodiment. In this second embodiment, aniterative procedure is also used to compute an approximate deconvolutionfilter, and the approximation is improved at each iteration until itsubstantially converges to the ideal solution. As aforementioned withrespect to the first embodiment, the level of convergence may bepredetermined or may be set by a user. The approximate deconvolutionfilter is denoted at iteration k as ŵ_(k)(n,m). At this iteration, theoutput of the deconvolution filter is:

$\begin{matrix}\begin{matrix}{{y_{k}\left( {n,m} \right)} = {{{\hat{w}}_{k}\left( {n,m} \right)}^{**}{r\left( {n,m} \right)}}} \\{{= {\sum{\sum{{{\hat{w}}_{k}\left( {i,j} \right)}{r\left( {{n\; - i},{m - j}} \right)}}}}};}\end{matrix} & {{Equation}\mspace{14mu} (13)}\end{matrix}$

The filter output may be written as the sum of the ideal term and theestimation noise as:

$\begin{matrix}\begin{matrix}{{y_{k}\left( {n,m} \right)} = {{{w\left( {n,m} \right)}^{**}{r\left( {n,m} \right)}} +}} \\{{\left\lbrack {{{\hat{w}}_{k}\left( {n,m} \right)} - {w\left( {n,m} \right)}} \right\rbrack^{**}{r\left( {n,m} \right)}}} \\{{= {{s\left( {n,m} \right)} + {v_{k}\left( {n,m} \right)}}};}\end{matrix} & {{Equation}\mspace{14mu} (14)}\end{matrix}$

where v(n,m) is the estimation noise, which is desirable to eliminate.An initial estimate of the correct image may be written as:

Ŝ _(k)(n,m)=ŵ _(k)(n,m)**r(n,m).   Equation (15)

However, this estimate may be iteratively improved. There are a numberof currently known techniques described in estimation theory to achievethis. A preferable option is the LMS algorithm. A block diagram of acalculation unit 80 which implements this method is shown in FIG. 8.

As an initial state, ĥ⁻¹ ₀(n,m) is set equal to μr(n,m). Then, thefollowing steps are iteratively repeated:

Step 1, an estimate of the correct image is calculated in a first 2D FIRfilter 82:

Ŝ _(k)(n,m)=ĥ _(k) ⁻¹(n,m)**r(n,m).

Step 2, a received signal based on the estimated correct image iscalculated in a non-linear estimator 84:

{tilde over (r)}(n,m)=g(Ŝ _(k)(n,m));

and the estimation error is calculated using an adder 86:

e _(k)(n,m)=r _(k)(n,m)−{tilde over (r)} _(k)(n,m).

Step 3, the inverse transfer function coefficients are then updated inthe LMS algorithm unit 88:

ĥ _(k+1) ⁻¹(n,m)=ĥ _(k) ⁻¹(n,m)+μr(n,m)e _(k)(n,m),

where μ is the step-size parameter.

The function g(.) calculated in step 2 is a non-linear function chosento yield a Bayes estimate of the image data. Since this function is notcentral to the present method and is well known to those of skill in theart, it will not be described in detail hereinafter.

There are known blind detection algorithms for calculating s(n,m) bylooking at higher order statistics of the image data r(n,m). A group ofalgorithms under this category are called Bussgang algorithms. There arealso variations called Sato algorithms, and Godard algorithms. Anotherclass of blind estimation algorithms use spectral properties(polyspectra) of the image data to deduce information about h(n,m). Anyappropriate blind estimation algorithm may be used to determine h(n,m),and to construct a correcting filter.

The first two embodiments of the present apparatus, system and methoddescribed hereinbefore correct blur in an image based on determining atransfer function that represents the motion of an imager while an imageis being captured, and then correcting for the blur by making use of the“inverse” transfer function. One method determines the transfer functionat the time the photograph is being captured by using devices that maydetect camera motion directly. The other method generates a transferfunction after the image is captured by using blind estimationtechniques. Both methods then post-process the digital image to correctfor blur. In both cases, the captured image is originally blurred bymotion, and the blur is then removed.

In accordance with a third embodiment, the blurring of an image isprevented as it is being captured, as described below. When an imager ismoved while an image is being captured, multiple copies of the sameimage are, in effect, recorded over each other. For example, when animage is captured digitally it is represented as pixel values in thesensor points of the image sensor. This is pictorially represented inFIG. 9, in which the imager (for example, a camera and its associatedlens) is not shown in order to simplify the depiction.

If the imager is shaken or moved while the image is being captured, thesituation is equivalent to copies of the same image being capturedmultiple times in an overlapping fashion with an offset. The result is ablurred image. This is particularly true if the shutter speed isrelatively slow compared to the motion of the camera. This isgraphically illustrated in FIG. 10.

When the shutter speed is sufficiently fast compared to the motion ofthe imager, blur does not occur, or is very limited, because thedisplacement of the imager is not large enough to cause the lightreflected from a point on the image to fall onto more than one point onthe image sensor. This third embodiment takes advantage of the abilityof an imager to record multiple images using fast shutter speeds. Whenan image is being captured using a setting of a relatively slow shutterspeed, the imager actually operates at a higher shutter speed (forinstance at the fastest shutter speed at which the imager is designed tooperate), and captures multiple images “back to back.”

For example, if the photograph is being taken with a shutter speedsetting of 1/125 sec and the fastest shutter speed of the camera is1/1000 sec, the camera actually captures 8 consecutive images, eachtaken with a shutter speed setting of 1/1000 sec. Then, the cameracombines the images into a single image by aligning them such that eachpixel corresponding to the same image point in each image is combinedpixel-by-pixel into one pixel value by adding pixel values, averagingthem, or using any other appropriate operation to combine them. Themultiple images may all be stored and aligned once all of them arecaptured, or alternatively, each image may be aligned and combined withthe first image in “real time” without the need to store all imagesindividually. The blur of the resulting image is substantially reduced,as depicted in FIG. 11.

The quality of an image may be measured in terms of signal-to-noisepower ratio (SNR). When a fast shutter speed is used, the SNR of theimage is degraded because the image sensor operates less effectivelywhen the amount of light falling on it is reduced. However, sincemultiple images are being added, this degradation is overcome. Indeed,an SNR improvement may be expected using this embodiment, because theimage data is being added coherently while the noise is being addednon-coherently. This phenomenon is the basis for such concepts asmaximal ratio combining (MRC).

To determine how to align the pixel values, a device that may detectmotion, such as an accelerometer or other motion sensor, is attached toor incorporated within the imager, and it records the motion of theimager while the photograph is being taken. The detected motionindicates how much the imager moved while each of the series of imageswas captured, each image having been captured back-to-back with a highshutter speed as set forth in the example above. The imager moves eachof the images in the series by an amount, which is preferably measuredin pixels, in the direction opposite the motion of the imager thatoccurred during the interval between the capture of the first image andeach respective image in the series. Thus, the shift of each image iscompensated for, and the correct pixels are aligned in each of theimages. This is illustrated in FIG. 11. The combined image will not beblurred since there is no spilling of image points into more than onepixel in the combined final image.

As an alternative to the third embodiment, the reference point foraligning the higher speed images is not the imager location, but thesubject itself. In other words, higher shutter speed images may bealigned and combined such that a designated subject in a field of viewis clear and sharp whereas other parts of the image may be blurred. Forexample, a moving subject such as a car in motion may be the designatedsubject. If high shutter speed images are combined such that the pointsof the image of the moving car are aligned, the image of the car will beclear and sharp, while the background is blurred. As a way to align adesignated subject, such as the car in this example, pattern recognitionand segmentation algorithms may be used that are well known to thoseskilled in the art, and defined in current literature.

Alternatively, a tracking signal that is transmitted from the subjectmay be used to convey its position. Alternatively, the user mayindicate, such as by an indicator in a viewfinder, which object in thefield of view is the designated subject to be kept blur-free.

Another embodiment compensates for movement of the imager or the subjectby adjusting the position of the image sensor during image capture,according to the inverse of the transfer function describing the imageror subject motion, or both. This embodiment is illustrated in FIG. 12.This embodiment is preferably used in digital cameras wherein the imagesensor 108 is a relatively small component and may be movedindependently of the camera, but may also be used with film.Accordingly, this embodiment makes use of motion sensors, and detectsthe movement of the camera and/or the subject while the image is beingcaptured. The signals from the motion sensors are used to controldevices that adjust the position of the image sensor. In FIG. 12,horizontal motion sensor 102 and vertical motion sensor 104 measuremovement of the camera while its shutter (not shown) is open and animage is being captured. The motion information is conveyed to acontroller 106, which determines and sends signals to devices 110 a, 110b, 110 c, and 110 d, which adjust the position of the image sensor 108.The control mechanism is such that the devices 110 a-d, for exampleelectromagnets or servos, move the image sensor 108 in the oppositedirection of the camera motion to prevent motion blur. Additionalsensors (not shown) may be used to detect motion of the subject, and thecontrol mechanism configured to correct for that motion as well.

FIG. 13 shows an example of a photographic image that is blurred due touser movement of the imager while taking the picture. FIG. 14 shows thesame image, corrected according to the present invention. The inventionsubstantially recovers the correct image from the blurred image.

Those skilled in the art will recognize that several embodiments areapplicable to digitized images which are blurred by uniform motion,regardless of the source of the image or the source of the motion blur.It is applicable to digital images blurred due to motion of the imager,of the subject, or both. In some cases, it is also applicable to imagescaptured on film and then scanned into digital files. In the lattercase, however, motion sensor information typically may not be available,and therefore only the blind estimation embodiment may be used. Also,where appropriate, the different embodiments of the invention may becombined. For example, the superposition embodiment may be used to avoidmost blur, and the correcting filter using blind estimation embodimentmay then be applied to correct the combined image for any remainingblur.

When the user selects a main subject in the viewfinder in the mannerdescribed above, the images captured by the imager are combined suchthat the main subject is blur-free at the expense of the background.Since this is post-processing and the sequential images are alreadycaptured and stored, the user may select another main subject to makeblur-free at the expense of the rest of the image and repeat the processfor the newly selected main subject as well.

An example application of another embodiment, where multiple images arecombined to create a blur-free image, is the all-digital implementationof a graduated filter. A graduated filter reduces the light intensityrecorded on the image sensor from parts of the subject. Whilephotographing a high-contrast subject, a graduated filter reduces thelight intensity from highlights, thereby reducing the dynamic range“spread” between the shadows and highlights. For example, whenphotographing a subject with a 5-stop dynamic range (a “stop” indicatinga move from one aperture setting to the next on a camera), a 2-stopgraduated filter that is used correctly may reduce the dynamic rangedown to 3-stops. An imager may mimic the function of a graduated filterby combining multiple images selectively as follows.

Every image being combined contributes to the light intensity of pixelsin the final image. By combining fewer images to form highlights thanthe number of images combined to form shadows, a final image is createdwhere the dynamic range between shadows and highlights is compressed.FIGS. 15A and 15B are an example of such a graduated filterimplementation—effectively attenuating the highlights in the upper partof the subject. FIG. 15A graphically demonstrates that when parts ofmultiple images 150, 152, 154, and 156, are combined, a verticallygraduated filter is created. FIG. 15B graphically demonstrates how thevertically graduated filter created in 15A reduces light intensity onthe upper portion of the image. Fewer images are added to create theupper part of the final image, resulting in reduced light intensity.

A user may select a preset graduated filter strength, and select adirection of graduation—for example, transitioning from darker (reducedlight intensity) to lighter from top of the image to bottom of theimage, or any other direction that the user may select. Since combiningof the images may be repeated with different parameters, the user maychange the graduated filter selection and obtain a different effectwithout having to take the picture (multiple pictures) again.

FIG. 16 shows an example selection of graduated filters that may bepresented to the user for selection, for example on a viewfinder or ascreen, either of which may be touch-sensitive. Graduated filter 160demonstrates that the darker area of the drawing depicts the lightintensity gradually increasing from top to bottom. Graduated filtersused in traditional photography are typically made out of glass and theyare circular in order to fit in front of a camera lens, moreover theless transparent part of the glass, which is intended to reduce lightintensity by allowing less light through, is darker as a result of beingmore opaque. In the depiction of a graduated filter 160-166, in FIG. 16an image that mimics the look of a glass graduated filter is used, sincethis is something familiar to photographers and hence intuitivelyrecognized. Therefore, the darker part of the filter represents lesstransparency, in other words stronger filtering effect. Light intensitygradually decreasing from top to bottom is demonstrated throughgraduated filter 162. Graduated filters 164 and 166 decrease light fromleft to right and right to left, respectively.

FIG. 17 shows another example selection of graduated filter orientationthat may be presented to the user. The user may rotate therepresentative graduated filter image to set the orientation of it, forexample on a viewfinder or on a touch-screen display. Separately or onthe same representative graduated filter image, the user may also selectthe “strength” of the graduated filter. The “strength” may be used tomean the amount of light that the digital filter blocks, or in otherwords, filters out. The stronger the filter, the less light intensityresults after the filter. The camera then applies the selected graduatedfilter orientation and strength when combining multiple images to formthe corrected image.

When an image is being captured that represents a dynamic range that islarger than what may be captured with the image sensor, parts of theimage become saturated. When this happens, information in the saturatedareas from the subject image is lost. This is shown in FIG. 18A. Lightintensity beyond a particular value cannot be represented due to thelimited dynamic range of the image sensor. In the actual image capturedall saturated pixels may appear white, for example. FIG. 18B representsthe case where a lower light setting, for example a faster shutterspeed, is used and no part of the image is saturated. However, thisimage is using only a portion of the dynamic range of the image sensorrepresented on the vertical axis in 18B. Similarly, FIG. 18C representsthe case where multiple images are combined, such as those representedin FIG. 18B, to form a final image that takes up the full dynamic rangeof the image sensor and avoids saturating the sensor.

Referring to FIG. 18D, a comparison of the case where a large dynamicrange subject saturates the image sensor, and the case where the samesubject is recorded in multiple images, for example using faster shutterspeed, and then combined selectively to form a final image that does notsaturate the image sensor is shown.

An alternative embodiment of a graduated filter in accordance with theteachings herein may vary graduation on a pixel-by-pixel basis, ratherthan in a particular direction. When combining multiple images to createa corrected image with a smaller dynamic range, a graduated filterimplementation may combine pixels from fewer of the multiple images forpixels representing highlights, while combining pixels from more of themultiple images for pixels representing shadows. In other words, thegraduated filter may work similarly to what is described above and inFIGS. 15A through 18D, except that the number of pixels being added maybe adjusted not based on the graduation pattern and graduation level asin FIG. 16 or FIG. 17 but based on the light intensity value of aparticular pixel, or groups of similar pixels. This has the effect ofcompressing the light intensity dynamic range on a pixel-by-pixel basis.This alternative implementation of the graduated filter works better forimages where the highlights and shadows of the subject are not localizedbut appear in multiple places in the image.

Alternatively, as shown in FIG. 19, instead of varying the number ofpixels combined, pixel values may be scaled to achieve the same effect.By scaling pixel values in highlights such that the light intensityvalue contributed by each added pixel is smaller, a combined image withcompressed dynamic range 198 is created. FIG. 19 describes this type ofa graduated filter. Multiple images 190, 192, 194 of the subject arecaptured, for example using a fast shutter speed, and then multipleimages are combined where the light intensity of pixels being combinedare scaled using a scaling matrix 196 having a plurality of scalingfactors x1, x2, x3, . . . x16. In the example of FIG. 19, the scalingfactors 196 are determined based on the light intensity recorded foreach pixel. For pixels representing strong highlights and thereforelikely to result in saturation of the resulting pixel when multipleimages are combined, a smaller scale factor is selected. For example,the light intensity may be represented as a number in the range of 0 to255. In the example shown in FIG. 19, each pixel of the multiple images190, 192, 194 will have a light intensity represented by a numberbetween 0 and 255, 0 representing the least and 255 representing thehighest intensity. In particular, 0 may represent black, complete lackof light, and 255 may represent white, complete saturation of light. Intraditional photography, parts of the image that exhibit lower lightvalues are termed “shadows”, and parts of the image that exhibit higherlight values are termed “highlights”. In the example shown in FIG. 19,if a particular pixel of image 190, for instance the pixel in the lefttop corner, has a light intensity value, say “23”, then even when addedup with the same pixel from the images 192 and 194, the final pixel mayrepresent a light intensity around “69”, which is well within the rangeof possible values. On the other hand, if a pixel in image 190 had alight intensity of 125, then once it is combined with the same pixelfrom images 192 and 194, it may represent a light intensity value around“375”, which is larger than the maximum value that may be represented,which is 255. In that case the pixel value in the combined imagesaturates, meaning reaches the maximum value of 255 and remains there.In the final image that pixel will be rendered as white and will nothave any details. However, if one applies the example embodiment herein,upon detecting that the light intensity of the left top pixel of image190 is “125”, a scaling factor of say x1=0.6 in the scaling matrix 196corresponding to that pixel may be applied. In that case, the lightintensity values of that pixel from images 190, 192, and 194 arecombined after scaling by 0.6. This would result in a light intensityvalue of around ‘225’(0.6*125+0.6*125+0.6*125), which is smaller thanthe maximum allowed value of 255. The corresponding pixel in thecombined image no longer represents saturation, but it renders ameaningful value. Note that in this example, for simplicity, it isassumed that the combining is an arithmetic addition and that the lightintensity of the left top pixels in all three images 190, 192, 194 areidentical, which may or may not be the case. Also in addition toarithmetic addition, one may devise combining methods that may involvelinear or non-linear arithmetic operations and functions. Above examplein intended to be illustrative, not limiting.

By adjusting the number of images being combined to form different partsof the resulting image, or by adjusting the scaling factors applied topixels being combined, one may adjust the strength of the graduatedfilter—which is the amount of dynamic range compression obtained in theresulting combined image. Below details are given on implementing twospecific graduated filters to demonstrate how to create such filters bycombining parts of multiple images selectively and by scaling pixelvalues.

FIGS. 20 and 21 show example implementations of a “top-to-bottom”graduated filter 206, selected by the user, where light intensity isdecreased gradually towards to top of the image.

In FIG. 20, pixels are scaled such that the rows of pixels towards thetop of the multiple images have progressively smaller scaling factorsapplied to them. Accordingly, in FIG. 20, x1<x2< . . . . Multiple images200, 202 and 204 of the subject are taken, for example using a fastshutter speed, and they are then combined according to the graduatedfilter 206 type selected by the user. Therefore, scaling factors x1<x2<. . . in order to lower the light value of pixels towards the top ofimage 208 are used. Hence, the top part of the final combined image 210is compressed gradually more since the top row is scaled using scalingfactor x1, and the second row is scaled using x2, where x1<x2, and soon. In FIG. 21, the same effect is created by combining fewer of themultiple images when forming the pixels towards the top of the image.Assuming the same type of graduated filter is selected by the user as inFIG. 20, when forming the pixel values in the final image, a fewernumber of images are combined for rows towards the top of the image,than for the rows toward the bottom of the image.

In FIG. 21, only one image 208(a) is used to create the pixels in thefirst row 214(a) of the final image 214, only two images 210(a) and210(b) are combined to create pixels in the second row 214(b) of thefinal image 214, three images 211(a), 211(b) and 211(c) are combined tocreate pixels in the third row 214(c) of the final image 214, whereasfour images 212(a), 212(b), 212(c) and 212(d) are combined to createpixels in the fourth row 214(d) in the final image 214. By combiningportions from more of the multiple images of the subject, there is ahigher light intensity combined in the final image 214.

A similar example where a right-to-left graduated filter is createdwhere light values are attenuated gradually more towards the right sideof the image is shown in FIGS. 22 and 23. In this case, the userselected a right-to-left graduated filter to be applied to the image. InFIG. 22, multiple images 220, 222, and 224 of the subject are taken andthen they are combined according to the graduated filter 226 typeselected by the user. Therefore, scaling factors x1, x2, . . . in orderto lower the light value of pixels towards the right in image 228. Image230 is created, which has a larger dynamic range and is not saturated.In FIG. 23, the same effect is created by combining fewer of themultiple images when forming the pixels towards the right side of theimage. Assuming the same type of graduated filter is selected by theuser as in FIG. 22, when forming the pixel values in the final image,fewer numbers of images are combined for rows towards the right side ofthe image. In FIG. 23, only two images 230 a and 230 b are combined tocreate pixels in column 2 of image 234, whereas four images 232 a, 232b, 232 c, and 232 d are combined to create pixels in column 4 in thefinal image 234.

It should be clear to someone skilled in the art that the alternativeimplementations of the graduated filter described above may be combined.For example, combining of multiple images may include combining ofdifferent number of images to form different parts of the correctedimage, while scaling of pixel values may also be employed at the sametime. It should also be clear that combining of multiple images may beperformed within an imaging device and the corrected combined image maybe generated by the imaging device, or the combining may be done outsidethe imaging device as a post process, such as on a computer or in thecloud. It should also be clear that combining of multiple images may beperformed after all of the multiple images for combining are captured,or alternatively multiple images may be combined one at a time as theyare each captured.

It should also be clear that combining of multiple images may beperformed multiple times with different graduated filter settings, suchas a user selecting a graduated filter configuration to obtain acorrected image and then changing the graduated filter configuration toobtain a second corrected image. It should also be clear that theimaging device may include other useful information about the image forpresenting it to the user, such as indicating the parts of the imagewhere pixel values are saturated because of strong highlights or deepshadows, thereby indicating how best to use the graduated filter. Otheruseful information for presentation to the user may include a histogramof the light strength values contained in the image.

In the aforementioned description, no distinction has been made betweenan imager that captures images one at a time, such as a digital camera,and one that captures sequence of images, such as digital or analogvideo recorders. A digital video recorder or similar device operatessubstantially the same way as a digital camera, with the addition ofvideo compression techniques to reduce the amount of image data beingstored, and various filtering operations used to improve image quality.The invention is also applicable to digital and analog video capture andprocessing, being applied to each image in the sequence of images, andmay be used in conjunction with compression and other filtering.

The implementation of the apparatus that performs the restoration of theimages to their correct form may be done as part of the imager capturingthe image, or it may be done as a post-process. When done as part of theimager, the image correcting apparatus may be implemented in anintegrated circuit, or in software to run on a processor, or acombination of the two. When done as a post process, a preferredembodiment is that the image data is input into a post processing devicesuch as a computer, and the blind estimation algorithm is performed by acomputer program. In this embodiment, the implementation could be adedicated computer program, or an add-on function to an existingcomputer program.

Where a computer program performs the image restoration, a blindestimation algorithm may be executed by the program to calculate theestimated transfer function h(n,m). Alternatively, motion informationmay be recorded by the camera at the time the image is captured, and maybe downloaded into the program to be used as an input to calculateh(n,m). In either case, the program then derives the correcting filterand applies the filter to correct the image.

It should also be noted that if there are multiple blurred objects in animage, and the blur is caused by the objects moving in differentdirections, the image of each object will be blurred differently, eachblurred object having a different transfer function describing itsmotion. The present invention may allow the user to individually selectindependently blurred parts of the image and individually correct onlythe selected parts, or alternatively, to correct a selected part of theimage at the expense of the rest of the image, resulting in ablur-corrected subject and a blurred background.

When increased accuracy is needed in obtaining h(n,m), those skilled inthe art will recognize that, in some cases, the motion information fromsensors may be used to calculate h(n,m), and an estimate of h(n,m) mayalso be calculated by blind estimation and the two transfer functionsmay be advantageously combined for more accurate results.

There are other signal processing algorithms and digital filters whichmay be applied to digital images in order to improve their colorsaturation, reduce noise, adjust contrast and sharpness, etc. These maybe incorporated as part of an imager, such as a digital camera, or aspart of a post-processing application, such as a photo editing softwarerunning on a computer. It should be clear to those skilled in the artthat those techniques may be applied in addition to the distortioncorrection of this invention.

What is claimed is:
 1. A method for use in an imaging device forcapturing an image of a high-dynamic range subject, the methodincluding: displaying a preview of the subject to be captured in adisplay of the device; capturing multiple images of the subject by thedevice, each image comprising parts of highlights and parts of shadows;determining a light intensity dynamic range for parts of the subject;determining number of images to be combined to form parts of a finalimage based, at least in part, on the dynamic range of the parts of thesubject; combining the multiple images to form a final image, whereinthe combining includes the determined number of images to be combined toform different parts of the final image; displaying the final image inthe display of the device; and storing the final image.